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ABSTRACT 

The report, which is part of a series, provides a 
summary of a 1985 meeting. The meeting was intended to (1) provide an 
overview of the Secondary Transition Intervention Effectiveness 
Institute's programs, (2) discuss how participants could be involved 
in the Institute's programs, and (3) develop a format for networking 
the results of federally funded projects to other persons concerned 
with the transition from school to work of youth with handicaps. 
Included in the document is the agenda from the meeting, an overview 
of the Transition Institute, the results of the discussions regarding 
program evaluation and technical assistance, an evaluation of the 
meeting, and a list of the participants and advisory members. Several 
appendixes include the research questionnaire, evaluation questions, 
technical assistance needs instrument, and the evaluation instrument. 
(DB) 
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The following principles guide our research related to the education and amployment of youth and 
adults with specialized education, training, employment, and adjustment needs. 



• Individuals have a basic right to be educated and to 
work in the environment that, least restricts their right 
to learn and interact with other students and persons 
who are not handicapped. 

• Individuals with varied abilities, social backgrounds, 
aptitudes, and learning styles must have equal 
access and opportunity to engage in education and 
work, and life-long learning. 

• Educational experiences must be planned, delivered, 
and evaluated based upon the unique abilities, social 
backgrounds, and learning styles of the individual. 

• Agencies, organizations, and individuals from a 
broad array of disciplines and professional fields must 
effectively and systematically coordinate their efforts 
to meet individual education and employment needs. 



• Individuals grow and mature throughout their lives 
requiring varying levels and lypss of educational and 
employment support. 

• The capability of an individual to obtain and hold 
meaningful and productive employment is important 
to the individuals quality of life. 

• Parents, advocates, and friends form a vitally 
important social network that is an instrumental 
aspect of education, transition to employment, and 
continuing employment. 



The Secondary Transition Intervention Effectiveness Institute is funded through the Office of Special 
Education Programs, Office of Special Education and Rehabilitative Services, U.S. Department of 
Education (contract number 300-85-0160). 

Project Officer: Dr. Mel Appell 



For more information on the Transition Institute at Illinois, please contact: 

Dr. Frank R. Rusch, Director 
Coliege of Education 
University of Illinois 
110 Education Building 
1310 South Sixth Street 
Champaign, Illinois 61820 
(217) 333-2325 
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INTRODUCTION 



This document overviews the proceedings from the first of five 
meetings sponsored by Special Education Programs, Office of Special 
Education and Rehabilitative Services, U.S. Department of Education 
(contract number 300-85-0160). These meetings are organized by the 
Secondary Transition Intervention Effectiveness Institute, College of 
Education, University of Illinois at Urbana-Champaign. A primary 
objective of the Transition Institute is to evaluate the effectiveness of 
model programs that are funded through the Office of Special Education ancf- 
Rehabilitative Services. Secondary objectives relate to (a) providing 
program evaluation technical assistance, (b) investigating transition 
issues through applied research, and (c) providing evaluation research and 
applied research experiences to graduate students in the area of 
transition programming. 

The meeting was held November 4-6, 1985 at the Lowes L'enfant Plaza 
Hotel 1n Washington, D.C. The purpose of the maeting was to (a) provide 
an overview of the Institute's programs, (b) discuss how participants 
could be involved in the Institute's programs, and (c) develop a format 
for networking the results of federally funded projects to other persons 
concerned with tht transition from school to work for youth with handicaps. 

This document provides a summary of the proceedings from the 
meeting. Included in the document is the agenda from the meeting, an 
overview of the Transition Institute, the results from the discussions 
concerning program evaluation and technical assistance, an evaluation of 
the meeting, and a list of the participants and advisory members. In 
addition, several appendices are presented which include the research 
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questionnaire, evaluation questions, technical assistance needs 
instrument, and evaluation instrument. 



Frank R. Rusch, Ph.D. directs of the Secondary Transition Intervention 
Effectiveness Institute. 
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OFFICE OF CAREER DEVELOPMENT FOR SPECIAL POPULATIONS 
University of Illinois at Urbana-Champalgn 

RESEARCH TRIANGLE INSTITUTE 

Research Triangle Park, North Carolina 



AGENDA 

SECONDARY TRANSITION INTERVENTION EFFECTIVENESS INSTITUTE* 

presents 

The Project Director's Annual Meeting 
November 4-6, 1985 
Washington, D.C. 

Monday, November 4 

8:00 - 9:00 REGISTRATION AND COFFEE 

9:00 - 9:15 WELCOME 

Patricia Guard, Director of Special Education 
Programs, Office of Special Education and 
Rehabilitative Services 

9:15-9:30 THE HISTORY OF THE FEDERAL INVOLVEMENT IN 

TRANSITION 

Nancy Safer, Special Education Programs, OSERS 

9:30 - 10:30 OVERVIEW OF TRANSITION INSTITUTE ACTIVITIES 

Frank R. Rusch, L. Allen Phelps, John 
Pyecha, Institute Staff 

10:30 - 10:45 BREAK 

10:45 - 12:00 INSTITUTE RESEARCH PROGRAM OVERVIEW 

National Research Needs Study 
Frank R. Rusch 

Social Ecology Research Project 
Jam's Chadsey-Rusch, Institute Staff 

12:00 - 1:30 LUNCH (on your own) 

1:30 - 2:00 INSTITUTE EVALUATION PROGRAM OVERVIEW 

L. Allen Phelps, Laird Heal, Lizanne 
DeStefano, Del Harnisch, Bob Stake 
Institute Staff 
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2:00 - 2:50 EVALUATION INPUT SESSION #1 



Group 1 - Evaluation Data/Information Base 
L. Allen Phelps 

Group 2 - Meta-analysis 
Laird Heal 

Group 3 - Student Assessment 
Llzanne DeStafano 

Group 4 - Extant and Longitudinal Data Bases 
Del Harnlsch 

Group 5 - Evaluation Research 
Robert Stake 



2:50 - 3:10 BREAK 

3:10 - 4:00 EVALUATION INPUT SESSION #2 



The concurrent sessions will be repeated 
and participants are free to choose the 
session they wish to attend. 

4:00 - 4:30 QUESTION/ANSWER SESSION 

L. Allen Phelps and Institute Evaluation 
Program Staff 

4:30 - 4:45 SUMMARY SESSION 

Frank R. Rusch and L. Allen Phelps 

4:45 - 6:30 SOCIAL HOUR 

CASH BAR 

Tuesday, November 5 
8:30 - 9:00 COFFEE 

9:00 - 10:00 INSTITUTE TECHNICAL ASSISTANCE PROGRAM OVERVIEW 

Joe Haenn, Institute Staff 
10:00 - 10:30 TECHNICAL ASSISTANCE NEEDS ASSESSMENT 

Anne Hocutt, Institute Staff 

10:30 - 10:45 BREAK 
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10:45 - 12:00 GROUP SESSIONS TO COMPLETE AND DISCUSS NEEDS 

ASSESSMENT 

Group 1 - Joe Haenn 

Group 2 - John Pyecha 

Group 3 - Anne Hocutt 

Group 4 - Ronald Wiegerink, Institute Staff 

12:00* - 1:30 LUNCH (on your own) 

1:30 - 2:00 TRANSITION: THE FEDERAL PERSPECTIVE 

Joan Standlee, Deputy Assistant Secretary 
for Special Education and Rehabilitative 
Services 

2:00 - 3:30 INFORMATION EXCHANGE SESSION BETWEEN OSERS STAFF 

AND PROJECT DIRECTORS 

Mel Appell Gary P. Lambour 

Betty Baker Richard P. Melia 

Judith Fein Joseph Rosenstein 

Patricia Guard Nancy Safer 

William Halloran Joan Standlee 

Carol Inman Paul Thompson 

Robert E. Jones Edv/ard Wilson 
Marty Kaufman 

3:30 - 4:00 EMERGING TRENDS AND ISSUES IN TRANSITION 

William Halloran, Special Education Programs, 
OSERS 

4:00 - 4:30 SUMMARY SESSION 

Frank R. Rusch and John Pyecha 

4:30 - 6:30 SOCIAL HOUR 

CASH BAR 

Wednesday, November 6 

8:00 - 8:15 COFFEE 

8:15 - 8:45 OVERVIEW OF TECHNICAL ASSISTANCE EVALUATION MANUAL 

John Pyecha 
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8:45 - 9:45 



9:45 - 10:00 
10:00 - 10:45 
10:45 - 11:00 



GROUP SESSIONS TO DISCUSS EVALUATION MANUAL 
Group 1 - Joe Haenn 
Group 2 - John Pyecha 
Group 3 - Anne Hocutt 

Group 4 - Ronald Wiegerink, Institute Staff 

BREAK 

GROUP REPORTS 
CLOSING 

Marty Kaufman, Division Director, Division 
of Educational Services, Special Education 
Programs, OSERS. 
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Overview of Transition institute 

The College of Education at the University of Illinois at Urbana- 
Ghampaign (UIUC) has received a federal contract to create an institute 
that will study and evaluate services delivered to disabled youth who are 
entering the job market. 

The Secondary Transition Intervention Effectiveness Institute, which 
will be funded for five years by the Office of Special Education and 
Rehabilitative Services (QSERS), US Department of Education, will be 
conducting research and working with federally funded secondary special 
education projects throughout the country. The Transition Institute is 
directed by Frank R. Rusch, professor of special education, and co- 
directed by L. Allen Phelps, associate dean of education and professor of 
vocational education . 

Recently, the United States Commission on Civil Rights (1983) 
reported that between 50% and 80% of all persons with disabilities are 
unemployed. These data suggest that a disproportionately large number of 
disabled persons do not obtain meaningful jobs. Several follow-up studies 
conducted in Vermont (Hasazi, Preskill, & Gordon, 1985), Virginia (Wehman, 
Kregel, & Zoller, 1984), and Colorado (Mithaug & Horiuchi, 1983) reflect 
similar figures. Based on these findings, it appears that—in spite of 
considerable recent attention on elementary and secondary education — 
meaningful employment outcomes for graduating students who are disabled 
have not materialized. 

Although several million individuals with disabilities in this 
country are denied, for various reasons, the opportunity to engage in 
meaningful employment, these individuals do possess the potential to live 
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and work 1n the community. These individuals have been the focus of 
attention by special educators, vocational educators, vocational 
rehabilitation personnel , adult service agencies, and many other agencies 
and organizations for the past three decades. Unfortunately, individuals 
who are mentally retarded, physically disabled, and/or otherwise disabled, 
have not made a successful transition to the community. Most of them 
either work in sheltered settings, are underemployed, or are unemployed 
and live with family, relatives, or friends without much hope of 
participating in their community in the manner most nondisabled persons 
participate. There is considerable evidence to suggest that these youth 
will not make any major gains in the world of work unless there is a 
concentrated effort to identify and introduce interventions that will lead 
to their employment. 

The Transition Institute is designed to address both the theoretical 
and practical problems of transition from school to work for youth with 
handicaps. The Transition Institute grew out of a consensus among 
legislative, professional, and advocacy organizations that an initiative 
was needed to establish a more systematic and effective delivery system tc 
assist youth with handicaps in making the transition from school or 
unemployment to work. The passage of Public Law 98-199 provided the 
authority to address this need specifically through Section 626, entitled 
"Secondary Education and Transitional Services for Handicapped Youth." 
The mission of the Transition Institute is threefold, "t will address a 
series of evaluation, technical assistance, and research activities. 
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Evaluation Research Program 

The Evaluation Research Program is directed by Dr. L. Allen Phelps. 
The evaluation program will collect and analyze data related to efforts to 
prepare students for employment. One of the major activities of the 
evaluation program will be to collect and summarize information about the 
model programs funded under the secondary and transition services 1 
initiative. Descriptive data from a variety of sources will be identified 
and summarized. Data will be collected annually on programs funded in the 
following categories: (a) service demonstration models, (b) cooperative 
models, (c) youth employment projects, (d) postsecondary projects, and (e) 
transition from school/institution to work. From the collected data, 
program profiles for each model program will be generated by the evalua- 
tion group. Following a review by responsible project officers and the 
Institute Advisory Committee, a Compendium of Program Profiles will be 
published and disseminated. 

As the evaluative information is collected, a series of meta- 
analyses will be undertaken to look at the comparative data on target 
audiences, objectives, and project progress to date. The meta-analysis 
approach, developed by Gene Glass and his colleagues at the University of 
Colorado-Boulder, Involves converting findings of each study to an effect 
size. The resulting product, a meta-analysis report, will be produced for 
dissemination to project directors, OSERS staff, and other key persons. 

A third major purpose of the evaluation program is to help model 
project directors select and use appropriate Instruments and procedures 
for assessing the entering and exiting skills of students. Following a 
survey of model project staff, a taxonomy of instructional objectives/ 
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student competencies and an inventory of assessment procedures will be 
developed. 

Two major products will emerge from this objective. The first is a 
Compendium Document (to be updated annually) to include the taxonomy, 
abstracts of project-produced and commercially produced assessment 
instruments, and examples of promising assessment procedures and 
instruments. A technical report will be prepared that examines content 
validity, reliability, and universality of the assessment procedures being 
used. 

The fourth major task of the evaluation program will entail examining 
the educational, employment, and independent living outcomes attained by 
handicapped youth as they exit school and enter the work force. Data will 
be collected to determine the extent to which transition is being 
successfully achieved by these students at the national, state, and local 
levels. 

A document modeled after the Digest of Data on Persons with 
Disabilities will be published annually describing the best available 
information on such variables as the incidence of handicapping conditions, 
employment and unemployment rates for both handicapped and nonhandicapped 
youth, minority status among handicapped youth, secondary school 
completion data, employment status, earnings, and residential 
arrangements. Two meta-analysis reports examining the follow-up studies 
on the post-school status of former handicapped and nonhandicapped 
students will also be included as products from this fourth objective. 

The final objective to be undertaken by the evaluation program 
involves evaluation research. The basic activities include: reviewing 
the evaluation literature; convening a group of recognized evaluation 
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theorists, project directors, and institute staff to delineate issues on 
two occasions; commissioning and synthesizing scholarly papers on key 
issues; and developing an agenda of needed evaluation research in the area 
of transitional services. 

Technical Assistance Program 
The Technical Assistance Program, housed at the Research Triangle 
Institute (RTI) at Research Triangle Park, North Carolina, will be 
directed by Dr. John Pyecha. This component's overall goal is to provide 
technical assistance on evaluation methodology to the federally funded 
secondary and transition model demonstration projects. 

In an attempt to develop the mutual trust necessary to encourage full 
utilization of technical assistance resources, the Transition Institute 
separated the technical assistance component from the functions of 
research and evaluation research program housed at the University of 
Illinois. The Transition Institute's Technical Assistance Center (TAC) is 
staffed with technical assistance specialists who have access to a 
comprehensive resource base of information, which includes resources in 
topical areas specific to the evaluation of secondary and transitional 
programs . 

The technical assistance delivery cycle consists of the following 
three distinct stages: program planning, program needs assessment, and 
delivery of technical assistance. During the program planning stage, TAC 
staff will review the evaluation plans of all of the federally funded 
secondary and transition model demonstration projects. Each evaluation 
plan will be analyzed according to a checklist containing such variables 
as the purpose of the plan, intended audience, questions and criteria, 
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evaluation approach, data collection procedures, analysis, and reporting 
strategies. In addition, a general purpose evaluation document will be 
formulated to address many of the same variables included on the 
check! ist . 

During the program needs assessment stage, the TAC staff will conduct 
a needs assessment of each funded demonstration project. The purpose of 
this assessment is to gather information regarding the types of technical 
assistance desired by individual projects, likely strategies for meeting 
these needs, and the anticipated level and types of technical assistance 
required (e.g., in terms of days, telephone calls, workshops) that match 
the resources allocated to the TAC staff. Information will also be 
gathered from the funded projects regarding the topics that should be 
covered in the general purpose evaluation document and the format that can 
best convey those selected topics. 

The third stage is the delivery of technical assistance. The 
technical assistance system consists of eight delivery strategies which 
consist of assistance being delivered at the project director's annual 
meeting, workshops, and other professional meetings, as well as through 
the telephone, electronic mail, materials, correspondence, and on-site 
visits. Type of technical assistance delivered will depend upon matching 
the needs of the individual projects with the resources allocated TAC. 

Applied Research Program 
The research program, directed by Dr. Frank R. Rusch, will adopt a 
systems-oriented approach to direct its activities. This approach 
contends that problems experienced by handicapped youth are a reflection 
of broader systems problems. Consequently, the problems require that the 
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Transition Institute seek solutions that reflect consideration of 
transition from a systems-analytic perspective, with interventions 
formulated at varying levels within the broader system. The research 
program will study and formulate interventions at four levels of 
analysis: the individual level, the small-group level, the community 
level, and the society level. 

At the individual level, interventions will be person-centered and 
thus will be directed toward ameliorating problems that handicapped youth 
may have, e.g., social skill problems and problems with generalization and 
independence. Problems that occur at the smal 1-group level are seen as 
emanating from difficulties that occur with primary groups, such as with 
the family and coworkers, and thus interventions will be aimed at changing 
the groups 1 behavior rather than the individual's behavior. At the 
community level, problems are viewed as the failure of organizations 
(e.g., the media and unions) to implement socially desirable values and 
goals, and consequently, the problems that need changing are within the 
organizations themselves. The fourth and final source of intervention 
strategies that will be studied will be at the societal level. At this 
level, interventions will be directed toward changing "institutions" 
(e.g., governmental organizations) with regard to social policy and 
economics. Although interventions will be formulated at varying levels of 
analysis, it is clear that interventions directed toward one level (e.g., 
individual) will often impact upon other levels (e.g., small group). 

Two tasks are presently being conducted by the research program. The 
first task consists of a two-phase research needs study. During the first 
phase, researchers at the University of Illinois generated a list of 
possible questions at all four levels of analysis (e.g., individual, small 
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group, community, and society) that were of interest to them and could be 
researched at the University of Illinois. These questions were then sent 
to 25 experts in the fields of special education, vocational education, 
and vocational rehabilitation for verification of the questions 1 
importance and for question revisions and additions. Based on the input 
of the 25 experts, the original questions have been reformulated (see 
Appendix A), 

During the second phase of the study, the newly revised questions 
will be sent to selected local and state leadership personnel in special 
education, vocational education, and vocational rehabilitation and to 
employers, parents, and directors of all the federally funded model 
demonstration transition programs so they can prioritize the questions. 
The results from this survey will be used in two ways. First, the results 
will be analyzed and disseminated to all interested persons. Second, 
researchers at the University of Illinois will be invited to submit 
research proposals that address the prioritized questions. These 
proposals will be reviewed by Dr. Rusch and the institute Advisory 
Committee. Short- and long-term investigation will begin in the summer of 
1986. 

In addition to the research needs study, the research program is also 
in the process of beginning a long-term investigation which will span five 
years of the Transition Institute's existence. This program will study 
the social ecology of the workplace. During the first year of the 
program, instrumentation and observation procedures that measure social 
relationships existing in the workplace will be designed and piloted. 
During the second and part of the third years, the objective of the 
program will be to study the social interactions and relationships that 
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exist between coworkers, between nonhandi capped and handicapped employees, 
and between coworkers with handicaps. The objectives of year 3 and part 
of year 4 will be to implement intervention studies designed to maximize 
the social fit between employees with handicaps and their work 
environment. These results will be disseminated during the fifth year of 
the program. 

Other Activities 

In addition to the three major programs of the Transition Institute 
(i.e., evaluation, technical assistance, and research), the Transition 
Institute will undertake a series of four other specialized, supportive 
activities. The first of these is to review and synthesize research 
related to evaluation methodology, efficacy of secondary and transitional 
services to handicapped youth, and topics that provide guidance and 
direction for the Transition Institute's evaluation, technical assistance, 
and research activities. Despite the present literature on school-to-work 
transition, much remains unknown about the nature of the transition 
process. With new literature continually being made available, the 
Transition Institute will identify and analyze these studies and findings 
throughout the contract period, prepare an annotated bibliography, and 
disseminate the results. 

The second specialized task involves conducting an annual meeting for 
project directors associated with model demonstration projects funded by 
OSERS each year of the five-year funding period. During the first 
meeting, an emphasis was placed on describing the Transition Institute's 
activities. During subsequent years, the emphasis will be placed on 
presenting research and evaluation findings, as well as facilitating 
information exchange among project directors and invited guests. 
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Communication between and among the funded projects and organizations 
related to the Transition Institute is essential. The third specialized 
task is the establishment and maintenance of ongoing communication 
mechanisms. Communication activities will occur between (1) OSERS and the 
University of Illinois at Urbana-Champaign (UIUC) and RTI , (2) UIUC and 
the model demonstration projects, (3) RTI and the model demonstration 
projects, and (4) the model demonstration projects themselves. A variety 
of communication mechanisms will be established to facilitate these 
interactions, including meetings, a handbook containing information about 
each project (e.g., project name and telephone number, type of project, 
goals and components, target population served), SpecialNet, and through 
the quarterly newsletter, Interchange . 

The fourth task to be accomplished by the transition Institute is 
that of providing research, evaluation, and technical assistance 
experiences to graduate students. During the first year, the Transition 
Institute will provide training experiences for more than 12 graduate 
students across each of the three primary programs. These students will 
work under the direction of the principal investigator, the institute co- 
director, and researchers associated with each program. 

Under the direction of the Institute faculty, graduate students will 
engage in seminars and field experiences focusing upon transitional 
programming, technical assistance, research, evaluation, and leadership 
and personnel preparation. These students will also support Transition 
Institute staff in such activities as: (1) reviewing and summarizing the 
literature on transitional programs, evaluation, and technical assistance; 
(2) developing and implementing research studies; (3) actively 
participating in evaluation and technical assistance activities; 
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(4) developing packages of materials in response to field requests; 

(5) participating in reviewing evaluation plans; and (6) making decisions 
on the most effective means of delivering technical assistance and 
conducting program evaluations. 

"Although the mission of the Transition Institute is to conduct 
research, evaluation, and technical assistance activities, we must not 
forget that these activities are designed to benefit all youth with 
handicaps who are engaged in the transition from school to the workplace. 
We also must not lose the momentum that was initiated by Madeline WiT; 
(Assistant Secretary for the Office of Special Education and Rehabilita- 
tive Services) and that has been maintained through the leadership and 
dedication from her staff. By working in partnership with the federally 
funded model demonstration projects and by documenting their successes, we 
will be able to make definitive statements about the transition process 
which will benefit all handicapped youth of the future" (p. 4) (Rusch, 
1985). 
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Perspectives on Program Evaluation 

This brief paper seeks to provide some initial definitions and 
perspectives about program evaluation held by the Evaluation Program 
staff. These definitions and perspectives help to frame a series of 
issues that are of interest to the planners and users of evaluative 
information generated by the Secondary Transition Intervention 
Effectiveness Institute. To some extent the paper will outline concepts 
to be woven into a philosophy of evaluation for the Institute. 

It is important for the reader to recognize the eclectic character of 
the Institute Evaluation staff. We come to t^e Institute activity with a 
highly diverse set of experiences, beliefs, alues, and philosophies about 
education in general and educational evaL en in particular. This 
diversity in staff selection was and will c^nt vje to be intentional, and 
is viewed as a major strength of our approach to the problems of 
evaluating secondary special education and transitional programs for 
handicapped youth. 

The following perspectives are offerod: 

1. Program evaluation, as all evaluation, is a search for good and 
bad; euphemistically strengths and needs. The search for the former has 
often overshadowed the search for the latter. 

2. Most program evaluation is done informally and intuitively. 
Some will be done formally, by design, with results made explicit and 
available for distribution. What makes for sound informal evaluation is 
not always sensible for formal evaluation, and vice versa. 

3. Program people have quite a bit to lose by participating in 
formal evaluation. The outcomes and criteria specified (even when agreed 
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upon) often will be quite poor representatives of the complexities of 
programs and the diverse needs of various stakeholders. It is doubtful 
that one can "count on the evaluation conveying what the program is al 1 
about." 

4. Evaluations and evaluation reports should be designed for 
explicit purposes and selected audiences. It is likely that federal 
officials, local school officials, parents, and other interested parties 
will have sufficiently dissimilar interests. Thus, it is not likely that 
all of their questions can be accommodated in a single report or with a 
common evaluation method. 

5. Research and evaluation are similar but different. Not only is 
evaluation value-centered, it focuses on a single object, e.g., a 
project. That object can be an aggregate, such as "transition 
demonstration projects," funded and unfunded. Research is a search for 
more general understandings, about general treatments, classes of 
projects, and populations of participants. The activities of the 
Evaluation Program will include evaluation, research, and evaluation 
research undertakings. 

6. Evaluation and research efforts put people at risk. Rules for 
research on the protection of human subjects need to be followed. 

7. Evaluation studies serve other purposes than informational 
purposes. They legitimate. They advocate. They explain. Principally, 
they help to improve programs and services. It is shortsighted to act as 
if the purpose was singular in focus or exclusively rational. 

8. It is not reasonable to suppose that the best evaluation study 
of a national program is the sum of the evaluation of its projects, or 
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that the best evaluative information for local use is what has been 
designated as needed for addressing national needs or national review. 

9. Generally, evaluation efforts in special education and 
rehabilitation have stressed pre-placement assessment and neglecting 
efforts to explore and describe the impact of programs upon individuals 
after placement. Further, it seems that post-placement evaluation has 
stressed short-term placement success at the expense of long-term success. 

10. The needs and interests of individual projects and programs are 
of primary importance in the Institute's evaluation milieu. Similarly, 
the rights and welfare of program clients supersede the interests of the 
agencies who serve them. 

11. This major federal commitment to program evaluation can and must 
be used to strengthen the knowledge base of educational and human service 
evaluation methodology. 
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Evaluation Data/Information Base 

The pool of responses to the Project Evaluation Information was based 
on 5, returned forms and the comments of 40 session participants at the 
Annual Meeting. The purpose of this task is to develop a computerized 
information and data base for the demonstration projects, develop profiles 
for individual projects and groups of projects, generate graphic displays 
of project data and information, and facilitate the use of the data base. 
This will be accomplished by developing preliminary data collection plans 
and instruments, pilot testing the instrumentation, making refinements, 
collecting core information from projects, summarizing the data and 
information, $nd developing profiles qnd displays (see Appendix B, Task 
6.1). 

Discussion Questions 

1. How can the Institute Evaluation Data Base be designed to be of 
optimal use to project directors? To federal agency personnel? What 
major concerns do you have about this data base? 

Responses and comments focused on the need to ensure that appropriate 
"hard" data are collected on outcomes for participants, on the types of 
interventions used in the projects, and on community characteristics. 
Projects should also be encouraged to share with others their "false 
starts." Profiles and successful and unsuccessful students/clients might 
also be developed. It was suggested that the data base contain 
comprehensive abstracts on each project arranged by competition. 
Newsletters and the Special Net bulletin board should be the means for 
sharing information from the data base. 
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2. How do you envision project directors using the Project Profiles 
developed annually from this task? What kinds of information should 
appear in these profiles? How do federal officials anticipate using these 
profi 1 es? 

Project directors reported they would most likely use the data base 
to network with other project directors with similar projects and 
interests. The prof iles' could also be used in response to inquiries about 
the need for such projects. Generally, participants felt sharing the 
profiles would enable directors to improve and strengthen their projects 
over time. 

3. To what extent is evaluative information being collected by 
projects different from that which they proposed in their grant proposals 
to collect? What is different about the current evaluation design from 
that which appeared in the original proposal (i.e., new purposes for 
evaluation, different audiences, alternative use of results, refocusing of 
project objectives)? If changes were made, what is the most efficient 
means for the Institute staff to determine the nature of these changes? 

Respondents here tended to be involved in the first year of their 
project and felt that: (a) it was too early to tell if changes were needed 
in their evaluation design, (b) their evaluation design was quite 
unsophisticated, or (c) they had come to the meeting expecting to be told 
which evaluation data to collect. 

4. If the evaluation section of your proposal was to be re-written 
at this point in the project, how would it be different? 
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Respondents and commentors indicated that they would be a bit more 
specific in their evaluation designs, that they might budget for an 
evaluator, and that they would emphasize the collection of hard data, 

5. If you are a project director/staff person, do you have 

available IBM or IBM-compatible personal computer? 60% Yes No 

If yes, does this computer have electronic communications capabilities? 

50% Yes No 

If yes, which communications software is used? SpecialNet was mentioned, 
but only by a few . 

6. What are the major questions that project directors are seeking 
to answer with the evaluation of their programs? In their view, which 
data or information best answers these questions? 

Among the responses were the following questions: Which interventions 
are most successful? What are the shortcomings of the field? Where are 
the gaps in the services? What are the appropriate roles of educators, 
parents, and others? What services have a long-term impact on students? 
What are the costs involved? What are the predictors of successful 
transitions? How effective is the supported work model on job placement 
and eventual success? How effective is the curriculum? What gains have 
been made by students? How can generalized outcomes be taught 
effectively? Can materials from other sources, such as the military, be 
adopted for use easily? How do programs train and place students in new 
and emerging occupations? 
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7. What are the major questions that federal officials are 
seeking to answer with the evaluation of the demonstration programs? In 
their view, which data and/or information provides the best answers to 
these questions? 

The most frequent notation here focused on the question of successful 
interventions to inform the federal government of future funding 
priorities. 

I am: 85% Project director 

Federal ;agency official 

15% Other, describe: coordinator, evaluator, etc. 
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Meta-Analyses 

The annual meeting was very helpful in establishing some realistic 
limits on the expectations that could be achieved in a meta-analysis based 
on the evaluation data from the projects that are to be reviewed by the 
Transition Institute. Two points are especially pertinent: first, no 
project had a control group, which is essential for a conventional meta- 
analysis (e.g., Glass, McGaw, & Srjiith, 1978); and second, very few 
projects were designed with the same objectives, making it perilous to 
aggregate results over projects. The strategy that the meta-analysis 
"te^m" will take in addressing these challenges is as follows. 

First, we shall compare similar projects that differ on one or only a 
few critical characteristics, so that projects can serve as controls for 
one another. For example, if 20 projects involve urban university- 
special education coalitions, and they differ only in that 10 have a high 
school workshop and the other 10 do not, then any different between these 
two groups can reasonably be attributed to the workshops. Second, we 
shall classify projects as carefully as possible, so that groups of 
similar projects can be formed for analysis purposes. This classification 
is crucial if comparisons are to be credible. 

Discussion Questions 

Five discussion questions were presented to two small group sessions 
at the annual meeting (see Appendix B, Task 6.2). The groups contained 
about 20 members each. The first group was assigned, and the second chose 
the meta-analysis session from the five that were available. The five 
questions will be shown here with a summary of the discussion stimulated 
by each. 
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1. What broad analyses of the procedures and outcomes of transition 
education demonstration projects would produce useful information for 
others in the field? 

2. What comparisons among projects would likely produce misleading 
i nformation? 

While the interaction was generally positive, the responses to 
questions 1 and 2 were largely negative. Participants were concerned that 
it would be very difficult to group projects because each one was unique 
unto itself. No two projects had the same procedures, and it would be 
unfair to compare their outcomes. 

The remainder of this report must be interpreted in terms of this 
initial reservation . 

The first issue raised under this topic was completely unexpected. 
The participants keyed on the term "useful information 11 in the question, 
and launched into a discussion of their greatest need: to determine a way 
to cement inter-agency agreements so that students are served by a 
cohesive service system rather than one that is splintered into guarded 
fiefdoms. Questions raised were: How can we work out agreements where it 
is clear who 1s responsible for what service and who is responsible for 
what payment? 

It is clear that this class of concern is beyond tlvi preview of the 
meta-analysis, but it seems reasonable for project directors to ask for 
information that will help them function more effectively. The Institute 
should probably develop an objective to deal with these issues in its 
renewal appl i cation . 
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3. How can projects best communicate their actual program 
procedures and results? 

- Original project grant proposals. 

- Director Interviews 

- Front line staff Interviews 

- Copies of progress reports 

- Surveys of staff 

- Others 

In their written responses especially, the respondents were most 
confident of director Interviews, somewhat confident of staff surveys and 
interviews, and least confident of grant proposals and progress reports. 
Respondents stated that grant proposals and progress reports are 
necessarily distortions of true activities and accomplishments, since any 
weakness is a threat to funding. 

4. What are the best ways to define subgroups or projects? 

- Project objectives 

- Student types 

- Program (e.g., profile regarding: assess, train, place, maintain) 

- Organization and management systems 

- Evaluation methods 

- Others 

Consistent with their response to question 1, participants were 
reluctant to agree that projects could be grouped. Nevertheless, they 
agreed that these were all reasonable ways to attempt the impossible. 
Other subgroup divisions were also suggested. As a refinement of 
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organization and management systems, respondents suggested that 
communities could be typed by size and history of supporting handicapped 
citizens. Furthermore, differences among service agencies must be 
considered 1n any classification of projects. 

5. What are the goals, objectives, and intended outcomes of the 
projects? 

Perhaps the greatest shock to the meta-analysis function of the 
evaluation section of the institute was the realization of the 
heterogeneity of the goals, objectives, and intended outcomes of these 
projects. In addition to the classification of projects who are actually 
engaged 1n the placement of students in out-of-school jobs, a major 
classification problem must be solved before information can be aggregated 
about those projects that have no such activity. Some of these have been 
established to develop a coalition to facilitate transition from school to 
adult living. Others address training but have no placement function. 

6. How are these intended outcomes assessed? Are the measures 
standard for each particular outcome? How can they be made more 
standardized? 

This question received Very little discussion. The participants 
indicated that they expected the Institute to help them in this regard, 
since their knowledge of assessment and measurement of program outcomes 
was limited. 
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Student Assessment 

This summary was compiled using seven written responses to the 
questions by conference participants and the comments of forty session 
participants at the annual meeting (see Appendix B, Task 6.3). The 
purpose of the student assessment task is to analyze and recommend 
instruments for measuring student characteristics, achievement, and 
adjustment. This will be accomplished by reviewing proposals, developing 
taxonomy of student skills taught and measures used, preparing a 
compendium of skills and assessment measures, producing a technical report 
analyzing validity, reliability, and universality of the measures, and 
disseminating a compendium/technical report. A synthesis of the questions 
1s presented below. 

1. What are the measurement priorities for projects? 

Most projects cited the use of student assessment as an aid in 

vocational training and transition planning. A secondary use is in the 

measurement of program outcomes as a measure of student progress and 
program effectiveness. 

2. To what extent can these priorities be met by commercial 
instruments? 

While the majority of the projects used commercial instruments as 
part of their assessment system, most supplemented it with questionnaires/ 
cheokl ists/surveys which have been developed specifically for the project. 

3. What is the relative importance of assessment for the following 
purposes: (a) Initial assessment for placement, (b) assessment for program 
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planning, (c) assessment during the training program, and (d) assessment 
for evaluation/outcome measures? 

Essentially all of the projects involved With direct service to 
students use assessment Information to determine eligibility for the 
program. After that, the most common use of assessment 1 ( s for program 
planning, I.e., assessment of vocational skills, social skills, adaptive 
behavior. Very few programs Include on-going assessment during training; 
1n those that do, it consists mostly of informal observation at the job 
site. Several projects assess consumer satisfaction or post-training 
skill levels as part of their evaluation plan. 

4. What are the major dissatisfactions with available measurement 
procedures for the purposes listed above? 

A number of projects commented upon the lack of standardized soda! 
skills instruments and career Interest Inventories for handicapped 
students. There was also a wide spread interest 1n the development of 
handicapped norms for some popular vocational assessment instruments. 

5. What employee/student information do employers/educators find 
most useful? 

In addition to information on vocational skills levels, projects also 
found information on work habits, social skills, learning styles level of 
support needed to function, academic skills, previous employment, and 
vocational strengths and needs useful. 

6. Is there a need to obtain summary information on measurement 
procedures being developed by other projects? What mechanisms are most 
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useful 1n accomplishing this task? What role should the Institute play 1n 
this process? 

Projects expressed an overwhelming need for some networking between 
projects regarding assessment procedures. It was generally felt that the 
Institute should take major responsibility for organizing and 
disseminating the information. A loose-leaf document, to be updated 
periodically, was the favored mechanism of dissemination. Possible Items 
to include in the document were a description of each Instrument, 
standardization Information, and availability. The document should be 
organized by skill measured. 

7. Have there been major changes in the plans for and uses of 
student assessment information since the proposal was written? What are 
they? Why were these changes made? 

It was difficult to compile responses to this question. The group 
discussion format did not allow many projects to report and only a small 
number of short answer forms were returned. It is hoped that more 
information will be obtained from the inclusion of this form sent to 
individual projects. 

8. How can the Institute be of greatest use to you 1n dealing with 
questions of student assessment? What changes 1n the proposed activities 
would you suggest? 

Summarizing and disseminating assessment Information from various 
projects was seen as the most useful task of the Institute. Some need for 
assistance in selecting, developing, and standardizing instruments was 
also requested. No sped fic. changes were suggested. 
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Extant and Longitudinal Databases 

The purpose of this task 1s to examine the educational, employment, 
and Independent living outcomes attained by youth with handicaps as they 
exit school and enter the workforce. This will be accomplished by 
conducting secondary analyses of primary longitudinal data sources and 
developing a plan for synthesizing follow-up studies examining the post- 
school status of special education students (see Appendix B, Task 6.4). 
The analysis to the questions below is based on eight responses. 

1. What 1s the nature and extent of education and employment 
outcomes achieved by youth with handicaps? 

Although the group had a good idea of employment/education conditions 
in their local geographic area, they were less aware of state or national 
figures, other than those commonly seen in OSERS releases. 

2. What selected demographic, educational, and vocational 
experiences are related to employment following graduation or exit from 
high school? 

A variety of experiences and characteristics were mentioned as 
contributing to employment. These included: (a) urban vs. rural, (b) type 
of handicapping condition, (c) type and extent of secondary and post- 
secondary training available, (d) extent of interagency coordination, and 
(e) community employment characteristics. 

3. What data sources and information resources are appropriate for 
describing youth with handicaps and examining some of the relationships 
between educational and employment factors? 
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Three sources were described to answer this question: (a) agency and 
st *%$ QOE files and data banks, (b) Bureau of Employment Statistics, and 

(c) studies conducted by special interest and advocacy groups, such as the 
Association for Retarded Citizens. 

4. To what extent are local, state, and national data sources 
useful, valid, and reliable in their estimates of educational, employment, 
and independent living status attainment? 

V(hile the issue of validity and reliability was not specifically 
addressed, participants indicated that an easy to use digest of these 
statistics would be valuable. 

5. What factors should be commonly examined in statewide and local 
follow-up studies of our special education students? 

Factors that should be examined include: (a) services available to 
students after graduation, (b) the types of students that use these 
services (e.g., by handicapping condition), and (c) employment outcomes. 

6. What information from transition studies would be most useful to 
educators and employers? 

Information listed by participants included: (a) type of client, (b) 
job characteristics, (c) instructional and maintenance strategies used, 

(d) support services required, (e) job duration, and (f) wages/benefits 
earned. 

7. What role should the Institute take in identifying the gaps in 
the information available with respect to secondary education and 
transitional services and activities? 
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The Institute should serve as a data bank for data on all of the 
projects: 

8. What procedures and summary displays are most helpful in 
describing youth with handicaps? 

A document with tables, charts, and graphs. 
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Evaluation Research 

A task of the Transition Institute evaluation group is to conduct a 
program of research on the design and practice of formal program 
evaluation of secondary and transition projects (see Appendix B, Task 6.5). 

1. Local evaluation information needs (mostly pursued informally) 
do not aggregate to constitute the national/professional needs for 
evaluative information. What are effective means for maintaining an 
awareness of this fact both locally and nationally? 

Concerning the issue of local evaluation information needs versus 
national information needs, the responses indicate a sensitivity to the 
idea that evaluation studies richly describe and analyze projects in such 
fashion that evaluation provides a service to projects by way of 
facilitating improved practices. While there was a sensitivity displayed 
in responses to the benefits of evaluation which are designed to seek 
maximum general izabi 1 ity, it was nonetheless believed that such evaluation 
designs do not contribute very much to our understanding of a project's 
idiosyncracies and processes of development. Evaluators would be prudent 
to be sensitive to both and to consider both levels in their planning. 

2. What is to be gained and lost by organizing our evaluation 
studies so as to conceptualize the local project as the "treatment having 
certain effects?" 

To perceive the evaluation studies as something which conceptualizes 
the local project as "the treatment having a certain effect," was felt to 
display an insensitivity to the distinctive aspects of transition 
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programs. In particular, the aspect of how such projects link schools 
with other agencies: e.g., community care institutions and local 
employment agencies. The question of linkage between various agencies 
which these projects effect was felt to be a desirable focus for 
evaluation studies. 

3. Given there is not enough brainpower to do the jobs that need to 
be done, how can we identify the higher priority evaluation tasks, 
avoiding front loading, being realistic about the burden associates and 
subordinates will accept, being realistic about how and how much 
evaluation data get used? 

In response to the issue of identifying the higher priority 
evaluation tasks, respondents showed a willingness to participate in such 
a task either by means of participating in a survey of evaluation needs 
from which priorities would be established and compared with evaluation 
plans of each project. An interest was expressed in meeting with 
interested project directors and evaluation personnel to discuss and share 
project evaluation methodologies. 

It was felt that Special Education/Transition programs were 
distinctive in the multiplicity of agencies and audiences which they 
address: school personnel, family, community agencies, and employers. It 
was felt that case studies would be very useful in treating some of the 
issues which surround attempts to have linkage between these agencies. In 
addition, it was felt that longitudinal studies focusing on handicapped 
people in employment should be considered in evaluation study. 
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TECHNICAL ASSISTANCE DISCUSSION RESULTS 



John Pyecha, Ph.D. 
Evaluation Technical Assistance 
Program Director 
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Synthesis of Responses to the Needs Assessment Instrument 

A total of 79 Needs Assessment Instruments were completed and 
returned to the Technical Assistance (TA) staff of the Transitions 
Institute by the December 1 cut-off date (see Appendix G for Needs 
Assessment Instrument). This represents a response rate of over 70 
percent of the transition projects funded by OSERS. It can be assumed, 
based upon an analysis of non-respondents, that most of the nonresponse 
occurred from projects which already have been completed or are almost 
completed. Therefore, the responses on which the subsequent analyses have 
been performed are fairly indicative of present needs for technical 
assi stance. 

Project Requests for TA 

There were 38 separate items on the Needs Assessment Instrument 
(including the "other" category for each of the six areas of evaluation) 
for which respondents could request technical assistance in the evaluation 
of their Transition Demonstration projects. The number of requests for TA 
ranged from a high of 29 for one project to a low of 0 requests for 20 
projects. In general, university-based projects and the more established 
(older) projects requested less technical assistance than the smaller and 
newer projects. 

Areas of TA R - luested 

Six of the l.> items for which TA was most requested are in the area 
of data collection. These items include determining methods for analyzing 
data; methods for organizing/storing/and reducing data; methods for 
insuring data accuracy and reliability; methods of data collection; 
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developing or identifying instruments or forms to be used in collecting 
data,, arid consideration about interpretation of results given design 
constraints. Other top items include selection of an evaluation approach 
or model determination of audiences' information needs, identification of 
resources and constraints regarding conduct of the evaluation, and 
determination of formats for reports to different audiences. 

Types of TA Delivery Desired 

Telephone and written correspondence are the types of technical 
assistance most requested for all areas (i.e., Basis of Evaluation Plan, 
Project Documentation, Design of Evaluation, Evaluation Management, Data 
Collection, and Reporting Evaluation Results) of evaluation. Telephone 
was rated as most helpful for Basis of the Evaluation, while written 
correspondence was rated most helpful for Evaluation Reporting. Workshops 
were rated as most helpful in the areas of Evaluation Design and Data 
Collection. It is in these two areas plus Evaluation Management where 
site visits were indicated to be most helpful. 

Timing for TA 

Requests regarding the Basis of the Evaluation will occur primarily 
over the next two months. While requests concerning Project Documentation 
and Evaluation Design will occur over the next five months. Requests 
about Data Collection and Evaluation Management will occur over the next 
eight months, with a peak between February and April. Requests about 
Evaluation Reporting will occur primarily during the summer, but will 
continue into 1987. 
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SpecialNet Utilization 

Based on the results, the use of SpecialNet for electronic mail 
purposes does not appear to be a high priority item. However, the 
electronic mail capabilities should be pursued for those projects which 
have expressed an interest. Approximately one-third of the respondents 
currently access SpecialNet. Further, 28 respondents would access a 
transition evaluation Bulletin Board Service if one were available, with 
another eleven projects undecided. 

Evaluation Workshop Topics 

The degree of interest expressed in each of 21 potential workshop 
topics was calculated based on different weights for different responses 
(e.g., a weight of 3 for "Of Much Interest," 2 for "Of Some Interest," 
etc.). The topics in which the most interested was expressed are: 
"Identifying potential evaluation areas for transition topics," 
"Evaluation for program improvement," "Evaluating change in handicapped 
populations," and "Measuring cost effectiveness." 

Discussions in the small groups about what professionals have found 
helpful in other workshops brought out a number of similar experiences and 
positions. In general, the project directors/staff attending the 
Transitions Conference thought that successful workshops on evaluations 
for transitions projects should: 

1. Focus on topics/subtopics of interest/applicability to groups or 
clusters of projects. Examples of groups/clusters could be different 
project types or projects serving specific populations of students. 
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2. Offer some combination of small group sessions on a specific 
topic/subtopic and opportunity for one-to-one help (TA staff with project 
staff), and balancing structured sessions with open sessions. 

3. Provide an opportunity for projects to share and discuss data. 

4. Use participants as experts. 

5. Provide advance agenda and materials and have defined outcomes/ 
products of the workshops. 

Sites for Workshops 

Of the 79 persons completing the Needs Assessment instrument, 58 
stated that they would be able to attend a workshop, 13 stated that they 
would not, 6 were undecided, and 2 did not respond to this question. 

Sites receiving the most checks indicating that project staff would/ 
could travel there were: Raleigh (N=30), Sain Francisco (N=25), Champaign- 
Urbana (N=23), New York (N=14), and Washington, D.C. (N=10). 
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Analysis of the Responses to the Needs Assessment Instrument 

A total of 79 Needs Assessment Instruments (NAI) were completed 
either during the Annual Meeting in Washington, D.C. in early November or 
were received by mail through a mail out following the Annual Meeting. The 
number and percent of instruments received from projects is summarized 
below by authorization number. 



Funding 


Instruments 


Completed 


Authority 


Number 


Percent 


84.023D 


8 


66.7% 


84.023G 


12 


80.0% 


84.078B 


11 


73.3% 


84.078C 


13 


92.9% 


84.128A 


3 


60.0% 


84.158A 


7 


43.8% 


84.158B 


7 


63.6% 


84.158C 


2 


28.6% 


85.158C 


16 


94.1% 


Total 


79 


70.5% 



Thus, the response rate was over 70%. It can be assumed, based upon 
an analysis of non-respondents, that most of the nonresponse occurred from 
projects which have already been completed or are almost completed. 
Therefore, the responses on which the subsequent analyses have been 
performed are fairly indicative of present needs for technical assistance. 

Project Requests for TA 

There are 38 separate items on the Needs Assessment Instrument 
(including the "other" category for each of the six areas of evaluation) 
for which respondents could request technical assistance in the evaluation 
of their Transition Demonstration projects. The number of requests for TA 
ranged form a high of 29 for one project to a low of 0 requests for 20 
projects. The number of requests was distributed as follows: 
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Number of 


Number of 


Requests 


Projects 


29 


1 


27 


1 


22 


1 


20 


2 


18 


3 


15 


2 


14 


2 


13 


2 


12 


4 


11 


3 


9 


5 


8 


3 


7 


5 


6 


4 


5 


2 


4 


4 


3 


5 


2 


5 


1 


5 


0 


20 



These requests for TA can be d1v1<M into the four levels of TA as follows: 



Level of TA 



Range of Number 
of Requests 



High (site visit) 15+ 
Mid (extended services) 8-14 
Low (limited services) 1-7 

None (telephone & BBS access) 0 (or no response) 



Number of 
Projects 

10 
19 
30 
53 



By funding authorization, these projects break out as follows: 



Level of TA High 



Mid 



Low 



None (0) No Response 



84.023D 
84.023G 
84.078B 
84.078C 
84.128A 
84.158A 
84.158B 
84.158C 
85.158C 



1 (8.3) 
1 (6.7) 

0 (0.0) 

1 (7.1) 

1 (20.0) 

2 (12.5) 
1 (9.1) 
0 (0.0) 

3 (17.6) 



0 (0.0) 
2 (13.3) 

2 (13.3) 

3 (21.4) 

1 (20.0) 

2 (12.5) 
2 (18.2) 
0 (0.0) 
7 (41.2) 



3 (25.0) 

3 (20.0) 
8 (53.3) 

4 (28.6) 
1 (20.0) 
3 (18.8) 
3 (27.3) 
0 (0.0) 

5 (29.4) 



4 (33.3) 
6 (40.0) 
1 (6.7) 

5 (35.7) 
0 (0.0) 

0 (0.0) 

1 (9.1) 

2 (28.6) 
1 (5.0) 



4 (33.3) 

3 (20.0) 

4 (26.7) 

1 (7.1) 

2 (40.0) 
9 (56.2) 

4 (36.4) 

5 (71.4) 
1 (5.9) 
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The above analysis confirms that university-based projects and the more 
established (older) projects are less in need of technical assistance than 
the smaller and newer projects. Even so, there appears to be a good 
distribution of projects by authorization across the proposed levels of TA. 



Types of TA Requested 

The top 15 types of technical assistance requested is demonstrated 
below: 



Area 


Item 


Number of 




Number 


Requests 


Rank 


Data Collection 


4 (Data analysis) 


34 


1 


Evaluation Design 


3 


31 


2 


Data Col lection 


3 


30 


3 


Data Col lection 


2 


29 


4 


Data Col lection 


5 


26 


5 


Evaluation Basis 


4 


22 


6 


Evaluation Management 


4 


22 


7 


Data Collection 


6 


22 


8 


Reporti ng 


4 


22 


9 


Reporting 


2 


20 


10 


Evaluation Design 


5 


20 


11 


Evaluation Basis 


5 


18 


12 


Data Col lection 


1 


18 


13 


Reporting 


6 


18 


14 


Evaluation Management 


1 


17 


15 



Obviously, the greatest need is in the area of Data Collection, with 
all items except item 7 (ethical problems) and item 8 (other). Other 
items of interest are evaluation design (item 3 of Evaluation Design), 
audience information needs (item 4 of Evaluation Basis), evaluation 
resources/constraints (item 4 of Evaluation Management), using evaluation 
results (item 4 of Reporting), targeting rfeports (item 2 of Reporting) and 
evaluation criteria (item 5 of Evaluation Design). 
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Types of TA Delivery Desired 

The requests for types of technical assistance breaks down as 
indicated below: 

Written 

Electronic Corre- Regional Site 
Tel ephone Mai 1 spondence Workshop Vi si t 



Evaluation (0.6) (0.2) (0.4) (0.2) (0.0) 

Basis [1.3/34] [1.0/14] [1.2/25] [1.4/9] [1.0/2] 

Project Docu- (0.4) (0.2) (0.4) (0.0) (0.1) 

mentation [1.4/25] [1.1/15] [1.4/23] [1.0/2] [1.5/4] 

Evaluation (0.5) (0.2) (0.4) (0.2) 

Management [1.3/27] [1.0/12] [1.2/25] [1.3/11] [1.6/8] 

Evaluation (0.5) (0.2) (0.4) (0.3) (0.3) 

Design [1.3/30] [1.1/14] [1.2/28] [1.4/17] [1.6/13] 

Data Collection (0.5) (0.2) (0.3) (0.5) (0.2) 

[1.3/30] [1.1/14] [1.3/28] [i:3/16] [1.7/10] 

Evaluation (0.5) (0.2) (0.5) (0.3) (0.1) 

Reporting [1.3/28] [1.1/16] [1.3/32] [1.5/13] [1.8/5] 



*5cdre$ are mean scores where 0 = no heed and 2 = most helpful; 
( ) = mean score for all respondents 

[ / ] = mean score/number of respondents, for respondents indicating a 
•T' (helpfui) or "2" (very helpful) 

Telephone and written correspondence are the types of technical 
assistance most requested for all areas of evaluation. Telephone is most 
helpful for Basis of the Evaluation, while written correspondence is most 
helpful for Evaluation Reporting. Regional workshops would be most 
helpful in the areas of Evaluation Design and Data Collection. It is in 
these two areas plus Evaluation Management where site visits are most 
helpful . s 
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Summary of Data for Evaluation Workshop 

Topics 

Seventy-nine persons responded to the twenty-one possible workshop 
topics listed in the Needs Assessment Instrument. A frequency count of 79 
means that all respondents checked one of the three response categories: 
"Of No Interest, 11 "Of Some Interest, 11 and "Of Much Interest." Frequencies 
of 78 and below mean that at least one respondent did not check any of the 
response categories. 

Data on the average response for each possible topic were calculated 
using 0 for nonresponse, 1 for the response "Of No Interest," 2 for 
"Of Some Interest, and 3 for "Of Much Interest." Thus, the higher the 
average response, the more interest expressed in a given topic. 

The number of persons responding to and the average response for each 
item are listed below: 





Possible Topics 


N 


Average 
Response 


1. 


Determine audiences for an evaluation (Stake 
holder Analysis) 


75 


1.7 


2. 


Determining what can/should be evaluated 
(Evaluabil ity Assessment) 


79 


2.2 


3. 


Describing the evaluation context 


78 


1.9 


4. 


How to develop an Evaluation Management Plan 


78 


2.2 


5. 


Identifying evaluation constraints 


79 


1.9 


6. 


Identifying potential evaluation areas for 
transition projects 


79 


2.4 
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Possible Topics 


N 


Average 
Response 


7. 


Identifying appropriate standardized instruments 


78 


2.1 


8. 


Developing questionnaires and survey instruments 


79 


2.2 


9. 


Choosing an evaluation design 


78 


2.2 


10. 


Ensuring accurate data collection 


78 


2.1 


11. 


Methods of data reduction/analysis 


78 


2.0 


12. 


Using the microcomputer for program evaluation 


79 


2.2 


13. 


Targeting evaluation reports 


78 


1.9 


14. 


How to write an evaluation report 


79 


2.2 


15. 


Utilization of evaluation results 


79 


2.1 


16. 


Presenting evaluation results to the public 


78 


2.1 


17. 


Evaluation for program improvement 


77 


2.3 


18. 


Evaluating change in handicapped populations 


78 


2.3 


19. 


Measuring cost effectiveness 


78 


2.3 


20. 


Types of test scores 


78 


1.5 



21. Sources of invalidity 



78 1.6 
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Discussions in the small groups about what professionals have found 
helpful in other workshops brought out a number of similar experiences and 
positions. In general, the project directors/staff attending the 
Transitions Conference thought that successful workshops on evaluations 
for transitions projects should: 

1. Focus on topics/subtopics of interest/applicability to groups or 
clusters of projects. Examples of groups/clusters could be different 
project types or projects serving specific populations of students. 

2. Offer some combination of small group sessions on a specific 
topic/subtopic and opportunity for one-to-one help (TA staff with project 
staff), and balancing structured sessions with open sessions. 

3. Provide an opportunity for projects to share and discuss data. 

4. Use participants as experts. 

5. Provide advance agenda and materials and have defined outcomes/ 
products of the workshops. 

Site 

Of the 79 persons completing the Needs Assessment Instrument, 58 
stated that they would be able to attend a workshop, 13 stated that they 
would not, 6 were undecided, and 2 did not respond to this question. 

Seventeen persons did not Indicate any city as being a site where 
they could/would attend a workshop. The remaining participants often 
checked more than one site where they would/could attend. Sites receiving 
the most checks indicating that project staff would/could travel there 
were: Raleigh (N=30), San Francisco (N=25), Champaign-Urbana (N=23), New 
York (N=14), and Washington, D.C. (N=10). Thirteen persons wrote on their 
instruments that they had no money to attend a workshop. 
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SpecialNet 

Fifty of the 79 respondents did not have access to SpecialNet, 21 did 
have access, 3 gave no answer, and 5 were undecided. When asked if they 
would access a SpecialNet bulletin board denoted to evaluation of 
transitions projects, 31 did not respond, 28 said yes, 9 said no, and 11 
were undecided. The 31 respondents likely included the 21 persons who 
stated that they already had access to SpecialNet since only those who 
stated that they did not have access to SpecialNet were to answer the 
question about accessing a bulletin board on transitions. Of note is that 
5 persons wrote in that they had no money to access SpecialNet. 

Summary and Recommendations 

Seventy-nine persons completed the Needs Assessment questionnaire. 
Of thfcse, it is possible that 64 persons (around 81%) may attend a 
workshop on evaluation of the transitions projects. Of these 64, 6 
(around 9%) were undecided, and it is reasonable to assume that their 
participation is a function of the content and format of the workshops and 
perhaps of available dollars. 

It was clear that if the workshops were held in Champaign/Urbana, 
Raleigh, and San Francisco, many, if not most, of the respondents could 
attend. Collectively, these sites received 78 checks from an unknown 
number of persons. The only other sites receiving 10 or more checks were 
New York (N=14) and Washington, D.C. (N=10). Given these results, it 
makes sense to conduct the workshops in Champaign/Urbana, Raleigh, and San 
Francisco. 

There was also some interest in SpecialNet. Assuming no overlap 
between the group of 21 who said they already had access to SpecialNet and 

o ... . . 
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the 28 who said that they currently do not have access but would be 
interested 1n accessing a bulletin board on evaluation for transitions 
topics, these 49 persons constitute 62% of the respondents. Addition of 
the 11 "undecided" respondents to this group brings the percentage of 
possible bulletin board users up to 76% of the respondents. On the other 
hand, the "certain" users — those who already have SpecialNet-— consti tute 
only around 26% of the respondents. The cost of setting up the bulletin 
board must be balanced against a guestimate of around 50-60% of the 
respondents who have or will get access to SpecialNet in the future. 

Based on the qualitative and quantitative data/information obtained 
at the Transitions Conference, there are a number of possibilities 
regarding the content and format of the workshops. It is certain that the 
workshops should address the most popular (and a frequently checked) 
topic: identifying potential evaluation areas for transition projects. 
Given the feedback about the importance of targeting/focusing evaluation 
materials/efforts on transitions projects, it is likely that targeting 
this topic for these projects enhanced its appeal. Perhaps part of the 
content for this topic is likely to be different for different clusters/ 
groups of projects (e.g., the evaluation areas for postsecondary projects 
might/might not overlap with those by cooperative projects) and thus 
satisfy project staff requests for focused small group sessions. 

Any additional content areas considered for the workshops should 
focus on the other three high scoring topics: "evaluation for program 
improvement," "evaluating change in handicapped populations," and 
"measuring cost effectiveness." Again, these topics are amenable to 
differentiation in content based on differences among projects in terms of 
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type of project or handicapped population served and thus would be 
appropriate for small group sessions. 

The format of the workshops should take into account the Conference 
participants wishes for a balance between open and structured, and large 
and small group sessions. Additionally, there should be opportunity for 
participants to discuss results and data among themselves and for short, 
one-on-one discussions with TA staff. 

Given this information, it is likely that three one and a half day 
workshops can be held in March, perhaps the first week of March (March 3- 
7) and the other two during the third week (March 17-21). The second week 
would ba devoted to analyzing feedback from the first workshop and to 
refining and improving the workshop based on this feedback. A "typical 11 
workshop might include (1) a one hour, 45 minute session on general 
strategies for identifying evaluation areas in transition projects; (2) 
one hour small group sessions on three other topics of interest; (3) one 
hour 45 minute small group sessions on the main topic, with the groups 
composing different project types; (4) a repeat of the one hour small 
group session. 
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Technical Assistance Plan and Schedule 



This plan summarizes the planned technical assistance activity for 
the project year by level of involvement. Input for the plan is drawn 
from discussions at the Annual Meeting, the administration and analysis of 
the Needs Assessment Instrument (see also "Analysis of the Responses to 
the Needs Assessment Instrument"), and interactions with projects through 
the provision of technical assistance. As per the Institute proposal, 
there will be four levels of activity. 

Level 4: Site Visit 

Ten projects will be eligible for this level of TA activity. 

Based on the Needs Assessment Instrument, those sites most in need of 

on-site TA are as follows: 

CFDR No . Project Location 

84.158A Los Alamos, NM 

84.158C Bowie, MD 

84.158B Dallas, TX 

84.078C Paoli, PA 

84.158A New York, NY 

84.158B Bismarck, ND 

84.158A Seattle, WA 

84.023D Moscow, ID 

These eight sites clearly requested the most assistance in terms 

of the number of requests for on-site TA by projects. These sites 

differ from the sites listed in the "Analysis" document because the 

earlier document did not cross-tabulate requests by type of TA (i.e., 

the above list includes those sites specifically requesting on-site 

TA for a substantial number (at least six) of specific areas of 

need). Since most of these projects are established projects in 

operation about a year, the requests were not in the area of 
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establishing the basis for evaluation or in project documentation. 
Rather, requests were most often for assistance in data collection, 
followed by assistance in design and management of the evaluation 
process. The timing for delivery of these services ranges from 
December 1985/January 1986 through August/November 1986, although 
most requests are for the periods of February /Apri 1 1986 and May /July 
1986. There is also a need for on-site TA assistance in reporting 
evaluation results, but this is generally during the period of 
August/November 1986 or sometime in 1987. 

These 8 projects will be given the first opportunity at 
receiving an on-site visit. An additional two projects and 
replacement projects (for those declining an on-site visit) will be 
selected based on those projects next in need according to their 
Needs Assessment responses and/or telephone requests for TA. Five of 
these projects requested assistance on at least 18 items on the Needs 
Assessment Instrument. 

Level 3: Telephone and Written Assistance 

Approximately 20 sites will be eligible for up to this level of 
technical assistance. This level includes extensive telephone and 
written assistance, including the development of project-specific 
documents, but does not include an on-site visit. 

Based on the Needs Assessment Instrument, there are about 21 
projects who could use this type of assistance. These projects 
requested assistance on from 8 to 14 of the items on the Needs 
Assessment Instrument. 
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Level 2: Limited Correspondence 

This level of assistance includes limited correspondence and up 
to two telephone responses for technical assistance. This includes 
all projects who requested assistance on one or more items of the 
Needs Assessment Instrument (about 30 projects in addition to those 
eligible for Level 3 or Level 4 services). 

Level 1: General Services 

Projects basically have self selected into this mode. These 
projects generally have sufficient evaluation resources available on- 
staff or locally with little need for assistance from the TA 
component. These projects will be eligible for all of the general 
services below. Based on responses to the Needs Assessment 
Instrument, there are approximately 50 projects at this level during 
this first year of assistance. This category includes projects which 
already have terminated (12 month projects) or will be terminating 
soon. 

General Services 

There will be five types of general services available to all 
projects. However, based on the Needs Assessment Instrument, 1t is 
anticipated that responses to the availability of these services will 
be distributed roughly according to the distribution of projects as 
defined by the four levels above. 

1- Annual Meeting : All projects are eligible to attend the 
annual meeting and participate in all of the sessions which are 
offered by the TA component. 
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2. Workshops : There will be a series of workshops offered 
each year. All projects will be eligible to attend and participate 
in these workshops. 

3. Telephone Hotline : A direct telephone line for TA 
assistance has been established at RTI only. TA component staff will 
accept collect telephone calls on this line (919/541-5900). An 
answering machine monitors this line at all times so that all calls 
will be received. All projects are eligible to use this hotline to 
request telephone assistance, although the level of assistance will 
be provided according to the four levels of service above. 

4. SpecialNet Bulletin Board : A bulletin board service 
(BBS) will be established on SpecialNet by the TA component. This 
bulletin board will provide an informational message which will be 
changed on a weekly basis and will be accessible by all projects who 
subscribe to SpecialNet for only the cost of the telephone call and 
access charges. SpecialNet also will be used for the exchange of 
messages between projects and between the projects and the TA 
component. One item on the BBS will be a listing of the SpecialNet 
user names which can be used to access Transition Demonstration 
Project personnel . 

5. Materials : As general materials are developed by the TA 
component they will be mailed to all projects. These materials 
include the general purpose evaluation document and all supplementary 
materials to that document. 
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EVALUATION OF THE ANNUAL MEETING 
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Synthesis of Evaluation Results 

Scope of the Annual Meeting 

The Project Directors 1 Annual Meeting, sponsored by the Secondary 
Transition Intervention Effectiveness Institute, was held November 4-6, 
1985 at the Lowes L 1 enfant PI aza Hotel in Washington, D.C. The purpose of 
the meeting was to (a) provide an overview of the Institute's programs, 
(b) discuss how participants could be involved in the Institute's 
programs, and (c) develop a format for networking the results of federally 
funded projects to other persons concerned with the transition from school 
to work for youth with handicaps. 

Invitations to attend the meeting were sent to all 112 project 
directors who had received federally-funded model demonstration transition 
grants, 29 persons who had federally-funded transition personnel 
preparation projects, and 15 OSERS personnel. Prereglstration commitments 
were received from 175 persons; in some cases, more than one person from 
each project elected to come to the meeting. 

Evaluation forms were included in a packet of materials that were 
distributed to all participants on the first day of the meeting. Of the 
195 persons attending the annual meeting, only 42 or 22% of the 
participants turned in their evaluation forms. This report presents the 
data from those evaluation forms and d^aws some recommendations for future 
annual meetings (see Appendix D). 

Participant Demographics 

The primary audience for which the annual meeting was intended was 
all the project directors who had received federally-funded model 
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demonstration transition grants. Of the 42 persons who turned in their 
evaluations, 16 (38%) described themselves as project directors. The 
remaining persons were university personnel (38%)*, project coordinators 
(21%), local administrators (10%), researchers (7%), state administrators 
(2%), and program evaluators (2%). 

Over half of the participants had their doctoral degrees (56%) and 
57% of the participants were special educators. The remaining 
participants were in the fields of rehabilitation (20%), vocational 
education (6%), and other (15%). Of those responding, 28% had 6-10 years 
of working experience, 23% had 3-5 years of working experience, 16% had 
either 0-2 years or over 16 years of experience, and 14% had 11-15 years 
of working experience. 

The majority of the participants were White (81%) and the remaining 
( i-mcipants were Black (5%), Hispanic (5%), American Indian (2%), and 
otnar (7%). Half of the participants were female and half were male. 
Seven percent of the persons attending the meeting had a handicapping 
condition. 

Evaluation Results 

In addition to determining the demographic information from the 
individual participants, the evaluation instrument was designed to probe 
participants 1 reactions to the organization of the meeting, the content of 
the meeting, and directions that future meetings should take. The items 
on the instrument were rated on a seven point Likert-scale format (range 



^Percentages that do not add up to 100% are due to people marking more 
than one category. 
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of responses were from 1-7), with additional space provided for comments. 
Quantitative and qualitative responses to individual items are reported 
and discussed below. 

Planning and organization of the meeting . On a seven point scale, 
participants rated the planning and organization of the meeting as a 4.6. 
Overall, participants believed that the meeting was well organized, 
particularly in light of the fact that the Institute had only been in 
operation approximately two months. Many participants mentioned that they 
would have liked more advance notice of the meeting which would have saved 
them money on airfares and accommodations (three persons commented that 
they thought the hotel was too expensive). In addition, one person 
mentioned that the meeting was probably held too early in the life of the 
Project. Participants also would have liked a more detailed agenda sent 
prior to the meeting. 

Participa nt expectations . In regard to the extent that the meeting 
format and agenda met their expectations, participants rated this item a 
2.9. Participants overwhelmingly believed that there was not enough time 
allocated to information sharing and networking among other participants; 
too much time was spent with the Institute staff talking. In addition, 
participants did not believe that the purpose of the meeting was clearly 
articulated. Several participants came to the meeting hoping to get 
"practical" ideas, but these expectations were not fulfilled (but this was 
not the purpose of the meeting, which again points to the need to clearly 
explain the purpose and agenda of the meeting). One participant was also 
disappointed that Madeleine Will cancelled. 

Institute research program overview . Participants rated the 
research program overview as a 4.0. Comments received regarding the 
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research program ranged from "very Interesting" to "too much," "too 
technical," and "too boring." Participants did not like the lecture 
format presentation and felt that much of the material should have been 
communicated via written materials. Several participants also commented 
that they did not understand how the research program would benefit them. 

Institute evaluation program overview . The evaluation program 
overview was rated a 3.8 by participants. While two persons commented 
that they believed the overview was clear, three persons felt the overview 
was unclear. Participants also commented that they thought the evaluation 
documents needed further clarification and that the presentations were not 
very interesting. In addition, participants believed that the roles they 
were to play and the roles the Institute was to play 1n regard to the 
evaluation of their project were unclear. 

Technical assistance program overview . The technical assistance 
overview was rated a 4.7. Participants believed that the overview was 
presented clearly, would be useful, and was a highlight of the meeting. 

Alternative evaluation perspectives . This question was designed to 
probe whether or not the participants were provided with information 
regarding alternative evaluation perspectives; the item was rated 2.8. 
Most participants believed that this Issue was not addressed. 

Technical assistance needs . This item was rated a 3.9. 
Participants believed that the needs assessment was very good, but that 
they were going to take a "wait and see" perspective on whether or not it 
would provide useful information to them. Two participants mentioned that 
because their projects had just gotten funded, it was too early to tell 
what their needs were. 
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Small group discussions . The small group discussions were rated a 
4.6. Participants particularly liked the student assessment session, 
needs assessment session, and meta-analysis session. Four of the 
participants liked the session with the project officers, while two 
participants did not. Two participants believed that the sessions were 
too large which prohibited interaction. 

Amount of information presented . Participants rated this item as a 
3.6. Eight persons believed that the time and type of information 
presented could have been better. People commented that there was too 
much dead time, that the information was not relevant, and there needed to 
be more time for small group discussion. Three persons believed that the 
amount of information presented was just right and well worth it. 

Information exchange . This item was rated a 3.6. Again, 
participants beiieved that not enough tirrie was devoted to information 
exchange between participants and that most small group sessions were too 
large to be useful. In addition, participants wanted to be broken up into 
homogeneous groups according to populations served or content objectives. 
Participants want more information exchange in the future. 

Overall rat ing of the meeting . The overall rating of the meeting 
was a 3.6. Whil& softe participants had negative comments (e.g., "should 
have had two days or 1 1/2 days max 11 and "would have liked smaller qroups 
with practical useful directions"), some participants had favorabW 
comments (e.g., "made me realize how fluch more our agency needs to do 1n 
assessment and evaluation," and "if what is assessed is delivered, it will 
be a great success; I think the commitment is there"). 

Negative features of the meeting . The negative features mentioned 
most frequently were;; :(1) presentations were too lengthy and too general; 
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did not like the general overview of the Institute; (2) not enough time to 
interact with other participants; (3) Madeleine Will's cancellation; 
(4) being talked "down 11 to; and (5) timing. 

Future directions . Participants overwhelmingly wanted more time 
for exchange/dissemination type of sessions where they could share 
projects and discuss common problems and strategies. They also wanted 
project updates at the second meeting. 

Concl usi on 

From the ratings and comments above, it is obvious that participants 
want specific information presented via a small group format. In 
addition, participants want more opportuni ties to exchange information 
with persons who have similar projects. The intent of the first meeting 
was to give an overview of the Institute and to get feedback from 
participants regarding their projects and their needs. This goal was 
accomplished; although relevant to the operation of the Institute, it was 
not relevant to the participants. The format for the second meeting will 
be to provide participants with more opportunities to exchange 
information, as well as opportunities to hear the results of updated 
Institute findings. This different format and information should be 
useful and relevant to second meeting participants. 
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LIST OF REGISTERED PARTICIPANTS 
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Secondary Transition Intervention Effectiveness Institute 
Participants 1 Roster 



Adams, Terence W. 
Appell , Mel 

Asselin, Susan B. 

Atkins, Robert 0. 
Auty, Bill 

Bailey, Susan 

Baker, Betty 

Baumgart, Diane M. 
Beam, Joyce K. 

Behle, Sue 

Bennett, Barbara P. 



Genesis Learning Center 
2305 Lebanon Road 

Office of Special Education 
and Rehabilitative Services 
Room 3523 Switzer Building 
330 C. Street, S.W. 

Virginia Polytechnic 
Institute and State Univ. 
Research Division 

Rehabilitation Institute 
3011 Baltimore 

University of Oregon 
Research & Training Center 
Center on Human Development 

Council of Chief State School 
Officers 

400 North Capitol St., N.W. 
Suite 379 

Office of Special Education 
and Rehabilitative Services 
Room 4630 Switzer Building 
330 C Street, S.W. 

University of Idaho 
Special Education Dept. 
Room 113 Education Bldg. 

Charles County Board of Educ. 
Project TET: Training for 
Effective Transition 
P.O. Box D 

Department of Social Serv. 
Division Services to 

Handicapped 
150W North Temple, #234 

Delaware Division Vocational 
Rehabil itation 

325 East 11th St., 4th Floor 



Nashville 
TN 37214 

Washi ngton 
D.C. 20202 



Blacksburg 
VA 

24061-3299 

Kansas City 
MO 64108 

Eugene 
OR 97403 



Washington 
D.C. 20001 



Washington 
D.C. 20202 



Moscow 
ID 83843 



La Plata 
MD 20646 



Salt Lake City 
UT 84103 



Wi lmi ngton 
DE 19801 
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Berkell, Dianne E. 

Berhacchio, Charles 
Bernthal , John E. 

Beun, Brian 

Brot/n, James M. 
Buckley, Jay T. 

Butterfield, Earl C. 
CahUl, Sheila 



Calder, Jr. , Clarence R. 



Callan, Jim 



Carter, J. E. 



Cat'apano, Patricia M. 



Chadsey-Rusch, Jam's 



Chambers, Marcia'A. 



Long Island University 

C.W. Post Campus 

Department of Special Education 

University of Southern Maine 
246 Deering Avenue 

University of Nebraska 
Postsecondary Intervention 
Model for LD 

202 Barkley Memorial Center 

Institutional Development & 
Economic Affairs Service, Inc. 
Magnolia Star Route 

University of Minnesota 
Room 460 Vo Tec Building 

University of Oregon 
Special izied Training Program 
130 Education 

College of Education 
322J Miller Hall, DQ-12 

The George Washington Univ. 
Office of Sponsored Research 
for the Department of Special 
Education 

2121 Eye Street, N.W. 

The University of Connecticut 

School of Education 

U-33 

University of Illinois 
270 Education Building 

Boston College - Project TRYAD 
McGuinn Hall B29 

Young Adult Institute 
460 West 34th Street 

University of Illinois 
110 Education Building 

National Center for Disabled 
and Vocational Education 
1960 Kenny Road 



Greenvale 
NY 11548 



Portland 
ME 04102 

Lincol n 
NE 

68583-0731 



Nederland 
CO 80466 



St. Paul 
MN 55108 

Eugene 
OR 97403 



Seattle 
WA 98195 

Washington 
D.C. 20052 



Storrs 
CT 06268 



Champaign 
IL 61820 

Chestnut Hill 
MA 02167 

New York 
NY 10001 

Champaign 
IL 61820 

Col umbus 
OH 43210 
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Cobb, Brian 

Cobb, R. B. 
Coker, Charles C. 
Collins, Terence G. 

Cook, Judith 
Cook, Lynn 

Crawford, Dorothy 
Cunningham, CI aire 

Curley, Sheri 
Dalke, Connie 

Davis, Earl 

Davis, Janice 

DeStefano, Lizanne 
Donnel lan, Anne M. 



Department of Special Educ. 
Social Work/Social Services 
405A Waterman Building 
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Research Questionnaire 



Please circle one number for each of the following 

What strategies do family and friends use to help youth with 
handicaps adjust to their jobs? Can the effective components of 
these strategies be isolated and combined to yield one strategy that 
can be taught to advocates/significant others in the work setting? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 

What rules can be formulated to guide teachers 1 systematic withdrawal 
of their instructional programs to facilitate students 1 independence? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 

What job conditions and/or incentives are most effective for 
increasing the likelihood that coworkers will (a) act as advocates, 
(b) participate in data collection, and/or (c) participate in 
training? 

Absolutely „ Moderately Not at all Don't 

Essential Important ' Important Know 

123456789 10 98 

What interest/interaction patterns exist among potential coworkers 
and how can this information be used to facilitate employment for 
youth with handicaps? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

Can transitional strategies that result in meaningful employment for 
the individual be used to facilitate recreational and residential 
adjustment? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 

What behaviors evidenced in social interactions are viewed as most 
negative by coworkers, supervisors, customers, or equally significant 
others within the work environment? What social behaviors are viewed 
as most positive by this group? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 
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7. In what ways have recently revised social sec? regulations (e.g. 
el igiblHty) produced significant changes 1n number of persons 
participating in income maintenance progr*:. < g., Supplemental 
Security Income)? 

Absolutely Moderately Not at all Don't 

essential Important Important Know 

123456789 10 98 

8. If students/youths are taught to evaluate social situations, what 
impact vill this have cn improving their social performance on the 
job? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 

9. What are the most appropriate roles and responsibilities for 
families, teachers, rehabilitation counselors, and vocational 
educators in the transition planning process? When should this 
process start? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 



123456789 10 98 



10. What intervention and collaborative strategies are most efficient for 
facilitating interagency cooperation, and how can these strategics be 
implemented at the local educational agency level? 

v>solutaly Moderately Not at all Don't 

Essential Important Important Know 

1 2 3 4 5 6 7 8 9 10 98 

11. What type of self-instructional package can students use to develop 
their independence on the job? What components of this package 
contribute most in accounting for students becoming independent? 

Absolutely Moderately Not at all Don't 

Essential Important - Important Know 

1 2 3 4 5 6 7 8 9 10 S3 

12. What naturally occurring social behaviors prompt other social skills 
in the workplace and how can we teach students/youth to respond 
appropriately to these cues? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 
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13. What social skills are necessary across work settings? If students 
do not possess these skills, how should these skills be taught or 
otherwise compensated for in the work environment? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 

14. What alternative work patterns (e.g., flextime, permanent part-time 
employment, and voluntary work) facilitate successful employment for 
persons with mild to severe handicaps, physical disabilities, etc.? 

Absolutely Moderately Not at all Don't 

Essenti al Important Important Know 

12345 o 789 10 98 

15. If general-case programming is used to teach vocational skills/ 
behaviors outside the work setting, how effectively will these target 
skills/behaviors carry over to actual work settings? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 

16. What social skill teaching strategies introduced in or* setting 
result in generalized performance in a secor; ' setting v *.g., 
simulated vs. natural, residential vs. employment, ir -tructional vs. 
noninstructional )? 

Absolutely Moderately V.;t at all Don't 

Essential 7mpo-- v»rt Important Know 

123456789 ID 98 

17. What are employers' an£ coworkers* attitudes regarding working with 
employees with handicaps and vice versa? Do these attitudes vary 
across handicapping condU ti on/ severity j job type, and prior exposure 
to the other group? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

1 2 3 4 5 6 7 8 9 10 98 

IP. How has the "state of the economy 1 * *>.i fuenced the nature of 

employment training programs offered to persons with handicaps? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 
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19. 



20. 



21. 



Who develops income maintenance program policies for individuals with 
handicaps? Upon what information base do they develop these 
policies? Which group or key individuals influence these policy 
makers? What interventions can be developed to influence these 
policy makers to formulate new guidelines that support independence? 



Absolutely 
Essential 
1 2 



Moderately 
Important 
4 5 6 



Not at all 
Important 
9 10 



Don't 
Know 
98 



What teaching strategies can be developed that change negative 
attitudes of coworkers and employers coward persons with handicaps? 
Do these changes affect overall community employment trends? 



Absolutely 
Essential 
1 2 



Moderately 
Important 
4 5 6 



8 



Not at all 
Important 
9 10 



Don't 
Know 
98 



How can coworkers be taught to assist in the training and evaluation 
of behaviors related to work? 



22. 



23. 



24. 



Absolutely 
Essential 
1 2 



Moderately 
Important 
4 5 € 



8 



Not at all 
Important 
9 10 



Don't 
Know 
98 



Can social skill training conducted in an employment setting increase 
positive interactions with coworkers and decrease negative 
interactions/ inappropriate behaviors, and if so, how? 



Absolutely 
Essential 
1 2 



Moderately 
Important 
4 5 6 



Not at all 
Important 
9 10 



Don't 
Know 
98 



What strategies are most effective for en sting parents' support for 
transition planning that focuses upon paid employment? 



Absolutely 
Essential 
1 2 



Moderately 
Important 
4 5 6 



8 



Not at all 
Important 
9 10 



Don't 
Know 
98 



How can employees with handicaps be integrated into social events, 
activities, and networks associated with work settings (e.g., off- 
site parties, athletic teams, spectator sports)? 



Absolutely 
Essential 
1 2 



Moderately 
Important 
4 5 6 



Not at all 
Important 
9 10 



Don't 
Know 
98 
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25. What attitudes are portrayed by key individuals in the local media 
regarding persons with disabilities? What effect do these attitudes 
have on employment? What strategies should educational/ 
rehabilitation agencies use to promote positive portrayal? 

Absolutely Moderately Not at all Don't 

Essential Important Important Know 

123456789 10 98 



Please supply us with questions you would like to see our research group 
address: 

1. 



2. 



3. 
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Task 6.1 Project Evaluation Information Base 



Overview of the Task: To develop a computerized information and data base 
for the demonstration projects; Develop profiles for individual 
projects and groups of projects; Generate graphic displays of project 
data and information; Facilitate the use of the data base. 

Plan of Operation: Develop preliminary data collection plans and instru- 
ments; Pilot test the instrumentation; Make refinements; Collect core 
information from projects; Summarize data and information; Develop 
profiles and displays. 

Discussion Questions 

1. How can the Institute Evaluation Data Base be designed to be of 

optimal use to project directors? To federal agency personnel? What 
major concerns do you have about this data base? 



2. How do you envision project directors using the Project Profiles 

developed annually from Task 6.1? What kinds of information should 
appear in these profiles? How do federal officials anticipate using 
these profiles? 



3. To what extent is evaluative information being collected by projects 
different from that which they proposed in their grant proposals to 
collect? What is different about the current evaluation design from 
that which appeared in the original proposal (i.e., new purposes for 
evaluation, different audiences, alternative use of results, 
refocusing of project objectives)? If changes were made, what is the 
most efficient means for the Institute staff to determine the nature 
of these changes? 



4. If the evaluation section of your proposal was to be re-written at 
this point in the project, how would it be different? 
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If you are a project director/staff person, do you have available IBM 

or IBM-compatible personal computer? Yes No 

If yes, does this computer have electronic communications 

capabilities? Yes No 

If yes, which communications software is used? 



What are the major questions that project directors are seeking to 
answer with the evaluation of their programs? In their view, which 
data or information best answers these questions? (Refer to the 
lengthy list of variables and outcomes.) 



What are the major questions that federal officials are seeking to 
answer with the eval uation of the demonstration programs? In their 
view, which data and/or information provides the best answers to 
these questions? 



I am: Project director 

Federal agency official 

, Other, describe: 



93 



Annual Meeti ng-1985 
90 



Task 6.2 - Meta Analysis 



Nature of the Task: The purpose of this task is to aggregate information 
from similar projects in order to attribute certain outcomes to 
certain interventions. In the early stages it is necessary to 
describe interventions as completely as possible and standardize 
outcome measures so that the results of similar projects can be 
combined and compared with others. However, more important than 
these technical necessities is establishing which questions are 
likely to be most useful to program practices in secondary and 
transition education and services. 

Discussion Questions 

1. What broad analyses of the procedures and outcomes of transition 

education demonstration projects would produce useful information for 
others in the field? 

a. comparison within "RFP competitions 11 

b. comparisons drawn from subgroups of transition education 
projects (see item 4) 

c. others 



2. What comparisons among projects would likely produce misleading 
information? 



How can projects best communicate their actual program procedures and 
results? 

a. original project grant proposals. 

b. director interviews 

c. front line staff interviews 

d. copies of progress reports 

e. surveys of staff 

f. others 
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What are the best ways to define subgroups or projects? 

a. project objectives 

b. student types 

c. program (e.g., profile regarding: assess, train, place, maintain) 

d. organization and management systems 

e. evaluation methods 

f. others. 

What are the goals, objectives, and Intended outcomes of the projects? 



How are these intended outcomes assessed? Are the measures standard 
for each particular outcome? How can they be made more standardized? 
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Task 6.3 - Assessment of Student Characteristics 



Overview of the Task: 

a. Development of a taxonomy of competencies and Instructional 
objectives. 

b. Review of commercial and project developed instruments. 

c. Production of a technical report analyzing the universality of 
assessment procedures used in the model projects, Including 
recommendations for improving assessment procedures and 
instrumentation. 

d. Dissemination of taxonomy/comp^rri u:,i/technical report document 
to project staff and Interests others . 

Discussion Questions 

1. What are the measurement priorities for projects? 



2. To what extent can these priorities be met by commercial instruments? 

3. What is the relative importance of assessment for the following 
purposes? 

a) initial assessment for placement 

b) assessment for program planning 

c) assessment du> ng the training program 

d) assessment for evaluation/outcome measures 



4. What are the major dissatisfactions with available measurement 
procedures for the purposes listed above? 
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What employee/student information do employers/educators find most 
useful? 



Is there a need to obtain summary information on measurement 
procedures being developed by other projects? What mechanisms are 
most useful in accomplishing this task? What role should the 
Institute play in this process? 



Have there been major changes in the plans for and uses of student 
assessment information since the proposal was written? What are 
they? Why were these changes made? 



How can the Institute be of greatest use to you in dealing with 
questions of student assessment? What changes in the 6.3 activities 
would you suggest? 
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Task 6.4 - Secondary Analysis of Educational and 
Employment Outcomes Data 



Nature of the Task: Examine the educational, employment, and independ- 
ent living outcomes attained by youth with handicaps as they exit 
school and enter the workforce; prepare summary tables/figures 
describing the reporting sources with data on youth with handicaps. 

Plan of Operation: Identify the data sources, the topical needs, and 
develop tables/figures accompanied by explanatory notes. Conduct 
secondary analyses of primary longitudinal data sources and develop a 
plan for synthesizing follow-up studies examining the post-school 
status of special education students. 

Discussion Questions 

1. What is the nature and extent of education and employment outcomes 
achieved by youth with handicaps? 



2. What selected demographic, educational, and vocational experiences 
are related to employment following graduation or exit from high 
school? 



3. What data sources and information resources are appropriate for 
describing youth with handicaps and examining some of the 
relationships between educational and employment factors? 



4. To what extent are local, state, and national data sources useful, 
valid, and reliable in their estimates of educational, employment, 
and independent living status attainment? 
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What factor* should be commonly examined in statewide and local 
follow-up studios of our special education students? 



What information from transition studies would be most useful to 
educators and employers? 



What role should the Institute take in identifying the gaps in the 
information available with respect to secondary education and 
transitional services and activities? 



What procedures and summary displays are most helpful in describing 
youth with handicaps? 
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Task 6.5 - Evaluation Research 



Nature of the Task: Study philosophy, design, and practice of formal 

program evaluation of secondary and transition projects; prepare and 
conduct program of research on that evaluation. 

Plan of Operation: Expand task force with consultants and correspond- 
ents; prepare issue papers; promote discussions; undertake and 
encourage research activities. 

Discussion Questions 

1. Local evaluation information needs (mostly pursued informally) do not 
aggregate to constitute the national/professional needs for 
evaluative information. What are effective means for maintaining an 
av/areness of this fact both locally and nationally? 



2. What is to be gained and lost by organizing our evaluation studies so 
as to conceptualize the local project as the "treatment having 
certain effects?" 



3. Given that there is not enough brainpower to do the jobs that need to 
be done, how can we identify the higher priority evaluation tasks, 
avoiding front loading, being realistic about the burden associates 
and subordinates will accept, being realistic about how and how much 
evaluation data get used? 



4. What is different about special education, and transition services 
particularly, that calls for pause in direct application of general 
program evaluation techniques? 



5. What key ideas about program evaluation are currently underutilized 
and should at least be tried out? 
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APPENDIX C 
NEEDS ASSESSMENT INSTRUMENT 
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NEEDS ASSESSMENT INSTRUMENT (A) 
A. Specific 

NAME: 

TITLE 

ADDRESS 



TELEPHONE NO. 



The purpose of this self-administered need assessment instrument is 
to obtain your perceptions about needs for technical assistance (TA) to 
help refine the plan and/or implementation of your project evaluation 
component. "Needs 11 1s defined as a discrepancy between what is and what 
is desired; thus, the Institute's TA staff are asking you to assess any 
discrepancies between your evaluation plan as it currently exists and as 
you would like to see it refined and/or implemented. Additionally, should 
you desire TA, this instrument assesses the type(s) of assistance you 
might want (telephone, regional workshops, etc.) and when you would want 
it. PI ease remember that the information obtained through this instrument 
is completely confidential and will not be shared with anyone outside the 
TA component of the Institute. 

The information obtained will be used to tailor TA to your individual 
needs. It also will be aggregated with data from other projects to help 
TA staff plan to deliver technical assistance in the most effective and 
efficient way possible. The TA staff thank you for providing us with this 
information. 
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INSTRUCTIONS 



This needs assessment instrument has been divided into six different 
areas pertaining to program -evaluation; each area has a number of specific 
items that might be considered during the planning/implementation of your 
project evaluation plan. In this instrument, you will be asked to assess 
the status of each and every item, your satisfaction with each item 
(including your desire to refine the item and, If you so desire, whether 
the revision will be done internally or by external TA staff), the type of 
technical assistance (TA) that you desire should you decide the external 
revision is best, and the time you think the TA should be delivered. 
Since it is unlikely that every item of your evaluation plan has been 
listed, there is space within each general evaluation area for you to 
write in any item(s) 1n your plan for which you need technical 
assistance. Respond to your written-in items exactly as to those already 
in the instrument. The specific rating process is discussed below. 

1. STATUS OF ITEM 

A check ( ) should be placed in the appropriate status level for 
each item. 

Not Applicable means that this particular item is not 
applicable to your project's evaluation plan. 

Does Not Exist means that this particular item may be 

applicable/relevp.-it to your evaluation plan, but that 
it is neither written down anywhere nor has it been 
discussed/specifically considered in the development 
of your project's evaluation plan. 
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Exists-Not Written means that this particular item is 
applicable/relevant and has been discussed or 
considered, but that nothing about it has been written 
down . 

Exists in Evaluation Plan means that this item is 

specifically addressed in the evaluation plan of your 

project's funding proposal. 
Exists Elsewhere moans that this item is addressed in 

writing some place other than in the evaluation plan 

of your project's funding proposal. 
SATISFACTION 

A check should be placed in the appropriate assessment of 
satisfaction for each item. 

All Right As Is means that this particular item in your 
project's evaluation plan does not need to be 
changed/ref i ned/modi f i ed. 
Revisions Done Internally means that this particular 
item does need changing/refinement, but that any 
changes can be done by project staff. 
Obtain Technical Assistance meafis that refinement of 

this item can best be accomplished with the help of 
technical assistance staff skilled in project 
evaluation. 
TYPE OF TECHNICAL ASSISTANCE DESIRED 

If you have determined that a given item should be improved with 
technical assistance, place a check ( ) under the type(s) of 
technical assistance that you believe is/are important to use in 
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addressing your need. More than one type of assistance can be 
checked or double checked for each Item. Pleas* note that due 
to financial restraints, only ten site visits can be made to 
projects during the coming year and that only three regional 
workshops can be held. However, 1f you believe that either of 
these types of assistance will be required, do not hesitate to 
Indicate this. 

Telephone means that your TA needs for this Item can be 
adequately addressed through telephone conversations 
with TA staff. 

Written Correspondence means that your TA needs for a 
given item require a more 

complicated/formal/structured response than can be 
accomplished through telephone conversations. 

Electronic Mail means that you have access to 

SpecialNet and that your TA needs for a given item may 
be met relatively quickly but on a more formal/ 
structured basis than a telephone call. 

Regional Workshop means that your TA needs for a given 

item/area are likely to be common to many projects and 
addressing your need; place two checks ( ) under the 
type(s) of technical assistance that you believe 
is/are very important or are complicated enough to 
warrant a workshop for groups of projects. 

Site Visit means that your TA needs for a given Item are 
applicable to many members of your staff, are likely 
to need a comprehensive/in-depth approach, and cannot 
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